The Canadian Light Source Control System: Lessons Learned from Building a Synchrotron and Beamlines Control System

The Canadian Light Source (CLS) is a 2.9 GeV synchrotron that recently went into service. This paper presents some of the choices made in the design of this facility control system, and lessons learned from the CLS project. Some of the more novel technology choices in the design of the CLS include: (1) fully open source based operating system (RTEMS and Linux); (2) extensive use of fibreoptics bridges to connect Industrial PC hardware to VME IO crates; (3) a design/drawing set based on the use of industrial process and instrumentation drawings applied to accelerator systems; (4) extensive use of Ethernet based PLC and other intelligent network devices; and (5) common control system infrastructure across the accelerator and beamline systems. INTRODUCTION The Saskatchewan Accelerator Laboratory (SAL) was established in 1964 and operated until the late 1990s. In 1999 the pulse stretcher ring was decommissioned and the accelerator was refurbished to become part of the injection system for the new Canadian Light Source (CLS) facility [1]. The CLS facility consists of a linear accelerator, which operates between 200 and 250 MeV, Booster Ring 1 (BR1) a 2.9 GeV electron synchrotron, Storage Ring 1 (SR1) a high-current 2.9 GeV electron storage ring, two diagnostic beamlines and seven synchrotron radiation beamlines used for experiments. The phase II expansion is now underway adding an addition seven beamlines while the phase III expansion is in the proposal stages. The scope of the CLS control system covers the accelerator, storage ring, building mechanical and beamline control functions and consists of over 50 000 data points. This paper discusses some of the principles that were used in the design of the control system and lessons learned by the design team. DESIGN PRINCIPLES Component Standardisation A strong emphasis was placed on commonality of components across the accelerator system and all of the beamlines. This not only applied to the control system but also the mechanical, vacuum and electrical systems. There are obvious benefits such as the reduction in spares required on hand. Less obvious are the significant cost savings resulting from reduced design costs as well as improved familiarity with the equipment. On the software side this translates into the need for fewer drivers and the ability to allocate additional resources to the drivers that are written. Software Engineering Process For mechanical design, Process Flow Drawings (PFD) are used to describe the desired machine optics and mechanical services. Process and Instrumentation Drawings (PID) are the created from the PFD in the early control system design stage. These serve as the main design input to the control system hardware design. The PID coupled with the requirements specification documentation serve as the main requirements for the software. PID are heavily used in the design of industrial control systems and are starting to gain acceptance in the accelerator community. These become fundamental to the system specification. For non-control software the Unified Process and Unified Modeling Language (UML) are starting to be adopted for newer projects. This includes the use of use cases to specify software behavior. The use of Computer Aided Software Engineering (CASE) tools within CLS are still in their early stages. MKS Source Integrity and Integrity Manager are used for version control and configuration management. PID drawings and wiring diagrams are done in AutoCAD while UML drawings are done in MS-Visio. CLS is beginning to adopt some IBM Rational CASE tools. Industrial PLC Systems A strong emphasis has been placed on the use of industrial programmable logic controllers (PLC) equipment. Telemecanique Momentum PLC are used extensively for equipment protection, mechanical services and building automation. Siemens S7/400 and S7/300 PLC equipment are used extensively in the booster ring RF, storage ring RF and cryogenics plant. EPICS EPICS is in use or planned for use in accelerator facilities in North America, Europe, Asia, the Middle East and Australia. This widespread adoption and acceptance of EPICS was a key motivation in selecting EPICS at the CLS. By selecting EPICS, CLS has benefited from being able to reuse a significant number of libraries, drivers, and application programs. EPICS core builds on contributions from various organizations, most notably Los Alamos and the Advanced Photon Source at Argonne. In addition CLS has reused the following EPICS extensions: (1) Extensible Display Manager (EDM) and the Channel Archiver from the Spallation Neutron Source (SNS); (2) a modified version of the Knob Manager from Los Alamos and (3) LabCA from Stanford. The Experimental Physics and Industrial Control System (EPICS) was initially adopted for the accelerator and beamline control systems. The use of EPICS was later expanded to include the building automation and mechanical services. A strong emphasis has been placed on using only one control system environment for all in-house developed and maintained systems. CLS has also limited the number of drivers and EPICS extensions in use. This strategy has allowed the relatively small controls team to be more responsive in their support role due to the familiarity with a common platform. The experience at CLS is that this strategy has been effective in managing the development and maintenance of the control system and controlling labour costs. Where this strategy has been problematic has been in developing acceptance by staff not involved in developing the control system, especially when using systems for which they are not familiar. One of the challenges faced in the use of EPICS has been compatibility of applications originating from other laboratories and configuration control. In several cases applications are written for a specific version or series of versions of EPICS. Care is required when integrating, testing and deploying these applications. These difficulties will hopefully be mitigated by the recent publication of the Channel Access (CA) Protocol and ongoing work by the developers of EPICS core to control changing and maintain backwards compatibility. CONTROL SYSTEM SERVERS Dell PowerEdge servers running RedHat Linux 7.2 or Scientific Linux are used to provide control network services. The main server is equipped with RAID disks, redundant power supplies and backed up by a hot-standby server. The dedicated servers provide network services, EPICS services, such as Process Variable gateways and channel archiving. Some file server functions are also provided by a Storage Area Network (SAN). The main server for the control network has been in continuous operation for nearly five hundred days; a hot standby server is also maintained. Operational experience suggests this to be a robust and reliable strategy. DATA NETWORK Several Virtual Local Area Networks (VLANs) are in use at the CLS running on a common hardware layer. The network is based on CISCO technology with redundant core switches. The network infrastructure is shared between the accelerator control system, beamline control, voice over Internet Protocol telephones, video and office networking needs. OPERATOR INTERFACES AND WORKSTATIONS Accelerator Operator Workstations A common control room is used for the accelerator and common mechanical services. Some of the old hardwired Linac control were initially retained and are now slowing being replaced. The control room was design to meet modern human factors engineering standards. In addition to also housing operator workstations, the control room houses a facility library containing system manuals, PID drawings and wiring diagrams for major accelerator and beamline systems. Operator Interface computers are provided in the control room with quad-screen displays. A limited number of lights, workstations and server equipment has backup emergency generator power. Beamline Operator Workstations Each beamline has a dedicated control area near the end-stations. These are equipment with Scientific Linux based workstations running similar OPI software as in the main control room. The EPICS Gateway and security software is used to limit beamline control of accelerator equipment and control of other beamlines while still providing remote monitoring capability. Display Manager and OPI Software After experimenting with alternatives, CLS selected the EDM display manager. By importing background graphics from the AutoCAD general arrangement drawings, CLS is able to produce realistic looking screens. Initially being window based, the design has been evolving towards screen based displays. Other OPI applications have been developed using Glade or Borland C++ Builder. Matlab Accelerator Toolkit Matlab is used as a platform to interact with the machine using several tool sets, these toolsets were the result of a collaborative effort between ALS and Spear over the past several years. Accelerator Toolbox (AT) [2] is used to model the storage ring lattice and perform simulations. LOCO [3] is an algorithm for debugging linear optics in the storage ring. The Middle Layer [4] is a set of scripts and functions that allow the accelerator tools above it to become machine and control system independent. Matlab Channel Access (MCA) [5] is used to connect the other tools to the EPICS control system. Matlab served as an indispensable platform during commissioning of the CLS. Many key measurements such as transverse orbit response and quadrupole centering were performed using this package. The accelerator tool box, LOCO, Middle Layer and MCA are open source. Matlab is of course a commercial product. Matlab was selected primarily to take advantage of the rich toolset for accelerator control that has been developed at ALS and SPEAR III. Matlab has since been applied as a scripting tool to aid in the commissioning of beamlines. Some of the beamlines have started to adopt Matlab as a general purpose scripting environment for beamline setup and scripting of experiments. CERN Root Still in the early stages, CLS has started to integrate CERN Root into the operator workstation software for beamlines. This provides for more advanced data plotting capabilities that commonly available in other EPICS user software as well as the potential for on-line real-time data analysis. INPUT-OUTPUT CONTROLLERS (IOC) Embedded Real-Time Output Controllers (EROC) Single board computers (SBC’s) based on the Motorola 68360 (originally developed for use at SAL) were chosen for use at the CLS [2]. Used extensively for control of RS-232 based devices, the EROC has four RS-232 lines. The RTEMS operating system is exclusively used on this hardware. RTEMS support existed early for these Input/Output Controllers (IOC), and is now used for IOC applications running on these SBCs. The earliest RTEMS based IOC was installed in 2000. Today there are in excess of 150 IOCs within the facility based on RTEMS. RTEMS has proven to be a very stable and mature platform. A version of RTEMS was selected for the Motorola 68360 based IOCs in 2000. However, the latest upgrades to EPICS (3.14 alpha1 to 3.14 beta1, and 3.14.7 is anticipated in the near future) have necessitated an upgrade to a newer version RTEMS. As it becomes more difficult to source comports for the original EROC hardware, CLS has started to prototype a PC104 based replacement using commercially available boards. The next generation of EROC contains 8 RS-232 lines. VME – Industrial PC Based IOC Connection to VME hardware at the CLS is not accomplished with a CPU board located inside each crate, rather a pair of fiber optic transceivers (1 PCI and 1 VME) [8] are used to connect a single PC computer to 1 or more VME crates. The fiber optic link is achieved by using a pair of cards SIS3100(VME) and SIS1100(PCI) installed into their respective back planes and connected with a fiber optic cable. Once connected, a Linux application is able to map the memory locations of any VME board installed in the VME crate into the application memory space of the Linux application. This allows the Linux application to access the registers or FIFO’s of a particular board installed in a VME crate simply by reading a pointer to a memory location. The actual transmission and reception of data across the fiber optic link is transparently taken care of by the SIS1100 PCI driver. Figure 1 shows how a single EPICS application running on a Linux IOC is able to read and write data to 3 different boards located in 3 different VME crates. /dev/SIS1100_2 (descriptor 2) VME CRATE 1: Hardware mapped memory VME CRATE 2: Hardware VME CRATE 3: Hardware Fiber Optic link